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Resumen: El objetivo de este articulo 
es presentar los resultados de un 
estudio piloto obtenidos a través de 
un cuestionario distribuido a futuros 
maestros de asignaturas de contextos 
AICLE y no AICLE que estan 
familiarizados con este tipo de 
metodologia por los estudios que 
estén realizando, ya sean de grado o 
posgrado. El articulo empieza con 
una introduccién al tema y continua 
con una revisidn de los estudios de 
investigaciOn relevantes en el campo 
de la evaluacién en este 4mbito. A 
continuacién, se presenta el método 
de nuestro estudio con informaci6n 
sobre el instrumento, los participantes 
y el contexto académico. 
Posteriormente, analizamos los 
resultados con ejemplos concretos de 
los datos obtenidos. Finalmente, 
reflexionaremos sobre los resultados 
obtenidos a modo de conclusién. Los 
resultados preliminares de_ este 
estudio piloto  mostraran los 
problemas mas importantes 
relacionados con la evaluacién en 
contextos AICLE de acuerdo con las 
expectativas de futuros maestros 
sobre como podria basarse_ la 


ensefianza siguiendo esta 
metodologia. 
Palabras clave: Evaluaci6n; 


Actitudes; AICLE; Estudio Piloto. 
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Abstract: The aim of this paper is to 
present a pilot study based on a 
questionnaire distributed to 
prospective teachers of CLIL and 
non-CLIL courses with familiarity 
on that type of teaching due to their 
own studies either through graduate 
or undergraduate courses. The paper 
opens with an introduction to the 
topic and continues with a review of 
relevant research studies in the field 
of CLIL assessment. Next, the 
method of our study is presented 
with information about the 
instrument, participants and 
academic context. After that, the 
results of the study are analysed and 
illustrated with examples from the 
data. Finally, different 
interpretations of the study will be 
discussed as a way of conclusion. The 
preliminary results of this pilot study 
will show the most significant issues 
involved in  CLIL assessment 
according to these prospective 
teachers’ expectations of what 
teaching following this methodology 
could be like. 


Keywords: Assessment; Attitudes; 
CLIL; Pilot Study. 
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ntroduction 


A brief research on assessment in Content Language Integrated 
Learning (CLIL) evidences a lack of references that specifically address 
this topic. This has in many cases been due to the overwhelming 
interest on its success (Coyle, 2006: 2) along with the need to justify its 
increasing use in many educational systems rather than on its 
assessment process (Jappinen, 2005; Pérez Cafiado, 2012). Indeed, 
many studies have already shown the beneficial effects that CLIL has 
had after its implementation in schools (Massler, Stotz & Queisser, 
2014; O'Dwyer & de Boer, 2015; Leal, 2016; Pérez Cafiado, 2017; De 
la Barra, Veloso, & Maluenda, 2018). However, few and very different 
models of assessment have so far been described (Morgan, 2006; 
Bertaux et al., 2010) and there are still elements in this respect that 
remain almost unconsidered such as teachers’ attitudes towards testing 
and assessment (Aiello, Di Martino & Di Sabato, 2017) especially in 
teacher education (Strotmann, Bamond, Lopez Lago, Bailen, Bonilla et 
al., 2014; Ennis, 2015). In this sense, as stated by Pérez Cafiado (2015: 
84), “evaluation is [...] vital in all respects, especially within a novel 
programme, to provide an insight into how students are reacting to new 
and unfamiliar circumstances”. It is also clear that the students’ positive 
attitudes are one of the cornerstones for educational and CLIL success 
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(ELDaou & Abdallah, 2019; Kirmizigul & Bektas, 2019; Tajgozari & 
Alimorad, 2019) especially in pre-service teachers (Genc, 2016). 
Therefore, the need for assessment measures to adequately evaluate 
both content and language becomes paramount (Diaz Cobo, 2009). In 
this respect, Pérez Cafiado adds (2015: 84) that in CLIL programmes “‘it 
is essential to determine if evaluation in the foreign language is carried 
out. Only then can we delve deeper to examine if communicative 
competences and content are being given priority and diversified 
evaluation models are adhered to.” Assessment can thus be considered a 
key aspect to continue with the correct implementation of bilingual 
programmes and the attitudes that teachers working in this environment 
have towards play a very important role. 


The aim of this paper is to present a pilot study based on a 
questionnaire distributed to prospective teachers of CLIL and non-CLIL 
courses with familiarity on that type of teaching due to their own 
studies either through graduate or undergraduate courses. Through the 
different questions of the questionnaire we wanted to find out what their 
attitudes toward it were. After this introduction, the paper continues 
with a review of relevant research studies in the field of CLIL 
assessment. Next, the method of our study is presented with information 
about the instrument, participants and academic context. After that, the 
results of the study are analysed and illustrated with examples from the 
data. Finally, different interpretations of the study will be discussed as a 
way of conclusion. 


1. Literature review 


In 2011 the European Centre for Modern Languages published 
the European Framework for CLIL Teacher Education (Marsh et al. 
2011). CLIL teachers in Spain are shaped by the requirements upon 
them before they begin to teach. These requirements are mostly related 
to their own language competence which may change from one region 
to another usually being Cl in the CEFR the requirement for English 
teachers and B2 for content subject teachers (of any other subject). The 
European Framework for CLIL Teacher Education has an extensive 
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reflection on teacher education providers being these universities or 
other agencies but the language requirement is usually covered by the 
demand of a Cl. However, there are other aspects that go unrevised 
such as the knowledge of basics in language and content methodology 
and many other aspects that can be found in other countries (design of 
material to use in the classroom, planning, etc.). For instance, Italy 
requires CLIL teachers “to attend and pass the specific methodological 
course provided by universities (known in Italy as ‘Corso di 
perfezionamento’)” (Aiello, Di Martino & Di Sabato, 2017: 72). In this 
sense, in Spain the situation is beginning to improve and in order to do 
so Teacher Training Centres and other institutions are working together 
to ensure that teachers working in bilingual programs are “updated in 
their language skills, methodological practices, materials development 
and assessment criteria, and that they are able to implement the 
European Language Portfolio at all levels”. (Milla Lara & Casas 
Pedrosa, 2018: 167). 


Further research has also approached assessment tools in CLIL 
contexts for primary education. Massler, Stotz & Queisser (2014) 
observed the kind of learner’s achievements in the relation of linguistic 
knowledge and the knowledge of content subjects in Primary 
Education. The authors also included their own approach of CLIL. 
However, their observational study just addressed two subjects and 
certainly not the teachers’ attitudes, which is currently one of the areas 
where more professional development is required in Spain. Likewise, 
Brevik & Moe (2012), observed that assessment in CLIL proved that 
the weakest students benefited best from assessment. However, Leal 
(2016) expressed that there is still a need to discriminate between 
content and language. This may be opposed to the views of certain 
linguists who believe that CLIL may not be different from English for 
Specific Purposes, which has been used for centuries (Garcia Laborda 
& Litzler, 2015), in terms of its use in “the teaching methodology, the 
course design, and the assessment procedure[s]” (Ennis, 2015: 358). 
Similar results were observed by Aguilar and Rodriguez (2012) who 
concluded that students in CLIL classes in university tend to focus on 
language and vocabulary gains but not on content in Spain. A more 
recent research study published in 2018 by Milla Lara and Casas 
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Pedrosa showed the outcomes of a qualitative study carried out with 
teachers in four provinces of eastern Andalusia (Spain) to analyse the 
development of CLIL programmes in this region. The authors 
concluded that teachers considered that in their CLIL lessons all 
contents were assessed and prioritized over linguistic aspects, and 
evaluation was diverse, formative, summative and holistic. Their results 
also showed that the oral components, even though they are part of the 
evaluation, are not highly considered, which is in line with what was 
stated by Lancaster (2016: 60), who reported that although teachers are 
generally satisfied with evaluation, “they admit an oral component is 
not always incorporated into assessment”. This has traditionally been 
the case of the Spanish educational system and despite an improvement 
in its situation, written skills are still prioritized over oral skills (Milla 
Lara & Casas Pedrosa, 2018). 


With the exception of this last-mentioned study, most research 
has failed to simplify and consider that assessment should mostly 
address language. Unfortunately centering the observation of learning in 
the language may lead to wrong assumptions about CLIL where the 
balance between language and content should prevail. This is in line 
with the definition of CLIL provided by Coyle, Hood and Marsh (2010, 
p. 1) as a “dual-focused educational approach in which an additional 
language is used for the learning and teaching of both content and 
language. That is, there is a focus not only on content, and not only on 
language”. Indeed, as stated by Kiely (2016), assessment in CLIL is 
complex due to the dual focus, which means that two assessment 
processes are involved, and a key issue is to what extent language and 
content are integrated on the assessment outcomes. As this author 
argues, this is not easy for the teacher, especially when it is a kind of 
methodology they have not been completely trained to deal with. 


In this sense, regarding teachers’ attitudes towards the 
assessment of CLIL, a first study was done by Morgan (2006) who just 
considered the difficulties of the different forms of assessment in CLIL. 
A second case was presented by O’Dwyer & De Boer (2015) who 
provided an approach to formative assessment through cooperative 
skills. Reierstam also conducted a study in 2015 in Sweden in which 


ISSN: 1988-8430 Pagina | 330 


Tejuelo, n° 31 (2020), pags. 325-341. Attitudinal Trends in CLIL Assessment: a Pilot Study 


she investigated teachers’ assessment practices in a CLIL context in 
three secondary schools. Her findings revealed that CLIL did not seem 
to have an effect on teachers’ assessment practices and that differences 
seemed to relate to their preferences or their perceptions of the subject 
they were teaching. In another study conducted by Vilkanciené & 
Rozgiené (2017) in Lithuania in the context of a CLIL project that 
aimed to upgrade the competences of subject teachers so as to integrate 
learning of content and language results revealed that, according to 
participants “CLIL remains one of the aspects that poses difficulties 
[...] and causes intense discussions even within the CLIL community 
itself’ (2017: 209). Their study also found that the lack of standards and 
guidance on assessment is considered to be problematic for teachers and 
that the development of material and training in this respect should be 
prioritized. 


Despite these studies, not even one has specifically developed 
teachers’ attitudes towards assessment. Therefore, it seemed to be area 
that needed to be further addressed in research and this was what we 
attempted to do with our pilot study as it will be shown in the next 
sections. 


2. Method 


In order to get insights into a topic that so far has been neglected 
in research, a survey was designed with seven questions (Appendix 1). 
This was done after conducting an extensive process of literature review 
and considering the criteria for the elaboration and application of 
instruments to collect data (McMillan & Schumacher, 2011). 


The participants of our study were both graduate and 
undergraduate students in language teaching courses at the University 
of Alcala and University of Jaén. A total of 35 students responded to the 
questionnaire. All the questions were closed-response except one about 
the problems they found in CLIL assessment. 
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The research questions of this pilot study were: 


1) What are participants’ opinions about assessment? 

2) What are the main problems and issues they found in 
assessing CLIL? 

3) Do they focus on content or language? 

4) Is writing a CLIL test more or less difficult than writing 


a language test? 


3. Results 


The following results show the answers to the above-mentioned 
questionnaire to analyze the perceptions of participants. The descriptive 
statistical data from the survey was also triangulated with qualitative 
information obtained from the open question. 


Regarding the first question, 85.7 % of participants stated that 
they were teaching CLIL courses, whereas 14.3 % said that they were 
not. As for the second question in which participants were asked if they 
had ever copied activities (even with minor changes) to write tests for 
classes other than English as it could be the case of Natural or Social 
Sciences, the following graph shows the collected answers: 
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Figure 1 
When I write tests for classes other than English (such as Natural or Social Sciences), 
do "copy" activities (with minor changes) that have already been done in class? 


@ Most times 
@ Sometimes 
@ Almost never 
@ Never 


Source: Own elaboration 


As shown in Figure 1, only 8.6 % of participants stated that they 
had never copied activities (with minor changes) when writing tests, 
whereas the highest percentage of participants (57.1 %) affirmed that 
they sometimes did and 14.3 % did it most of the time. This shows that 
most participants did not design their own assessment tools and had to 
resort to existing material to evaluate their students’ learning. 


Regarding the difficulty that writing tests for these subjects 
entails, participants’ answers were the following: 


ISSN: 1988-8430 Pagina | 333 


Tejuelo, n° 31 (2020), pags. 325-341. Astitudinal Trends in CLIL Assessment: a Pilot Study 


Figure 2 
I consider that writing a test to assess subjects other than English (such as Natural or 
Social Sciences). 


@ is very difficult 

@ is just about the same as for 
English 

@ is easier than for English 


Source: Own elaboration 


However, Figure 2 shows that for more than half of the 
participants designing a test for a CLIL context is just about the same as 
for English and only 20% found it very difficult. In this regard, these 
were the main difficulties that the participants of our pilot study 
indicated when preparing exams for CLIL students. Answers have been 
grouped in different thematic categories of analysis: 


Table 1 

Participant’s answers regarding difficulties when preparing exams for CLIL students 
Topic Answers 
Students’ level of English 8 
Finding the right topics 4 
Design of an exam according to students’ needs 2 


Source: Own elaboration 


Regarding students’ level of English, one participant stated that 
it is harder to know your students’ level of English in a Science class 
because they are not used to writing complete paragraphs in English 
and another one stated that he would be worried that students would not 
understand the questions. Another participant said that CLIL and non- 
CLIL students could not be assessed following the same level of 
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difficulty and another one said that the difficulty would be related to 
being clear enough on the content so that students would not be 
confused. Another participant’s answer added that when developing 
tests of this kind questions that require to be developed could not be 
used. Again, this shows that there is a sense of confusion among the 
criteria and method that needs to be followed to design a test in a CLIL 
context, which is in line with what Vilkanciene & Rozgiene (2017) 
stated in their study regarding the difficulties this poses for teachers. 


As for the importance of language in CLIL or bilingual 
Education, these were the answers that participants provided: 


Figure 3 
Importance of language in CLIL of Bilingual Education 


@ More importance than 
content 


@ the same importance as 
content 


@ Less importance than content 


Source: Own elaboration 


As shown in Figure 3, the highest percentage (74.3 %) 
corresponds to participants that considered language as important as 
content, whereas only 8.6 % considered language to be more important 
than content. This situation seems to have improved when compared to 
the results obtained by Aguilar and Rodriguez in their study from 2012, 
which concluded that students in CLIL classes in Spain focused more 
on language and vocabulary and not on content. Therefore, our 
participants’ answers seem to be in line with the proper balance of 
content and language for what is considered to be a successful 
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implementation of a CLIL subject according to Coyle, Hood and Marsh 
(2010). 


Regarding the use of alternative tools for assessment such as 
portfolios, presentations or others, 71.4 % of participants stated that 
they use both tests and other tools of assessments and only 5.7% of 
participants indicated that they only use tests and the same percentage 
chose the option of “I do not use tests”. 


Finally, when asked if they considered systematically (and 
numerically) their observations in class, the majority of participants 
answered affirmatively (82.9 %) (Figure 4). However, only 40 % of 
them indicated that they assigned these observations a grade. 


Figure 4 
Do you also consider systematically (and numerically) your observations in class? 


@ No, | do not 


@ Yes, but | do not assign them 
a grade 


@ Yes, | assign them a grade 


Source: Own elaboration 


Conclusions 


The aim of this paper was to present a pilot study based on a 
questionnaire distributed to prospective teachers of CLIL and non-CLIL 
courses with familiarity on that type of teaching due to their own 
studies either through graduate or undergraduate courses. In the section 
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dedicated to the theoretical framework of our study, we provided a 
review of the recent literature that focused on assessment in CLIL 
contexts. As was stated, despite the existence of studies that focus on 
how teaches currently assess their students and some of the difficulties 
they encounter, we could confirm that there is no research that 
specifically develop teachers’ attitudes towards assessment in this 
respect. Therefore, the four research questions that were formulated 
allowed us to obtain some preliminary results on the topic. First of all, 
regarding participants’ opinions about assessment, we can conclude for 
more than half of the participants designing a test for a CLIL context is 
just about the same as for English and only 20% found it very difficult. 
Second, the main problems they find in assessing CLIL are related to 
how they can assess students’ level of English, even though they tend to 
focus equally on content and language (third question). Finally, writing 
a CLIL test usually not design their own assessment tools and resort to 
existing material to evaluate their students’ learning. In accordance with 
the analysed literature, our answers support the idea that more 
importance should be givenat the university level on the importance of 
assessment in CLIL contexts. Therefore, efforts should be made to 
ensure the right combination of language and content in assessment 
practices and this is what training for prospective teachers in CLIL 
contexts should try to aim to for its successful implementation. Despite 
the limitations of this pilot study, we believe that we have gained a 
preliminary insight into this topic and that we have obtained some 
results that allow us to open a line for future research in such a 
neglected area. 
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Appendix 1: Assessment in CLIL Survey 


Question 1: Iam teaching CLIL courses 
Yes 
No 


Question 2: When I write tests for classes other than English (such as Natural or 
Social Sciences), I “copy” activities (with minor changes) that have already been 
done in class: 

Most times 

Sometimes 

Almost never 

Never 


Question 3: I consider that writing a test to assess subjects other than English (such 
as Natural or Social Sciences) 

Is very difficult 

Is just about the same as for English 

Is easier than for English 


Question 4: Point the main problems that you consider having when preparing exams 
for CLIL students 


Question 5: In CLIL or bilingual Education, language has 
More importance than content 

The same importance as content 

Less importance than content 


Question 6: Do you use alternative assessment tools (portfolios, presentations, others) 
Yes, I do not use tests 

No 

No, I only use tests 

Yes, I use both tests and other tools of assessment 


Question 7: Do you also consider systematically (and numerically) your observations 
in class? 

No, I do not 

Yes, but I do not assign them a grade 

Yes, I assign them a grade 
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